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ABSTRACT 



The goal of a project was to design, test, and evaluate 
several computer programs that allow students in introductory literature and 
poetry courses to explore a poem in detail and, through a dialogue with the 
program, to develop their own interpretation of it. Computer programs were 
completed on poems by Robert Frost and W.H. Auden. Both programs were tested, 
revised, and retested in classes at Kingsborough Community College, New York, 
and at other institutions. Preliminary versions of two more computer programs 
were produced- -one on Gerard Manley Hopkins' "The Windhover" and one on 
elementary prosody. Feedback from students and faculty was overwhelmingly 
positive and was used to implement changes in the software. Statistical 
results were inconclusive, but seemed to indicate that those who benefited 
most from the programs were able to apply what they had learned to the study 
of another related poem. Appendixes contain information for the Fund for the 
Improvement of Postsecondary Education; statistical results of testing in 
year 1 and year 2; student questionnaires; a report by outside evaluator Dr. 
Mohamed Tazari; a grid used for programming; print -outs of student sessions; 
and a conference program. (RS) 
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Exploring Poetry Through Interactive Computer Programs 
Kingsborough Community College of the City University of New York 
2001 Oriental Boulevard 
Brooklyn, NY 11225 

Project Directors: Howard Nimchinsky and Jocelyn Camp 
(718) 368-5939 

Executive Summary 

Project Overview 

The purpose of our project was to design, test, and evaluate several distinct software 
programs, each on an individual poem. Each program would allow students in introductory 
literature and poetry courses to explore the poem in depth and develop an interpretation of it. 
With the help of a group of faculty, with whom we consulted throughout the project, we chose 
the poems to be used, and in the first year of the grant tested preliminary and revised versions of 
the first program, on Robert Frost's "Design," and completed the preliminary version of a second 
one, on W.H. Auden's "Musee des Beaux Arts." In the second year we presented our work at 
three conferences, tested and revised "Musee des Beaux Arts" extensively, continued planning 
and began the programming for two more programs (one on Gerard Manley Hopkins' "The 
Windhover," and one on elementary prosody) and hosted a conference on "Computers and the 
Humanities" at our college. Students at a community college and a senior college used the 
programs during the two years of the grant in a total of eighteen poetry or introductory literature 
courses, and several other classes served as control groups. Many suggestions made by these 
students and teachers were implemented in later versions of the programs, as well as those of an 
outside evaluator on educational software design. 

Purpose 

The problem our project addressed was the difficulty that students in introductory 
literature classes often have with close reading of poetry, and their reluctance to do so. Our goal 
was to emphasize close reading and analysis. During the course of the grant our understanding of 
the problem itself did not fundamentally change. In fact, the ways the students used the programs 
made us aware of reading difficulties we had not anticipated. The records of each use of the 
programs indicate how students go about reading and analyzing; this taught us more about their 
learning processes than we could have discovered through traditional classroom instruction. 
According to the data we collected, the programs we designed were for the most part successful 
in focusing attention on the language of the poems and allowing students to develop their own 
reasoned interpretations. 

Any administrative pitfalls involved in using our software would have to do with providing 
the requisite technology to support it. Technology is developing and changing at such a rapid 
pace that both the software designer and the campus must constantly be prepared to adjust. The 
college must be responsible for keeping hardware up to date, an expensive proposition in days of 
tightened budgets. Teachers need some instruction as well, though the programs are really 
self-explanatory and "computer literacy" is not required. 
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Background and Origins 

At Kingsborough, all English faculty teach developmental as well as literature courses. 

Our FIPSE project originally developed out of our work as reading teachers. In the late 1980's 
we designed a developmental reading exercise, programmed in Prolog, in which students would 
answer questions on a short passage of text and the computer would respond. With the help of a 
series of small grants we wrote five of these programs and used them in reading courses. We 
realized that the same basic techniques could be applied to literature, and determined to create 
several programs in the interpretation of poetry. 

Support from the College has been essential to our work on the FIPSE grant. One 
necessity for the success of a project like ours is the cooperation of the academic computing staff 
of the college, which, for the most part, we had. They helped in setting up and supervising labs, 
giving us access to equipment (such as scanners), and allowing us to put updated versions of our 
programs on the network. The College provided computer and printer upgrades for the Project 
Directors' offices, access to the Toolbook hotline, and some travel funds. We were dependent 
upon our cooperating faculty from Kingsborough and other campuses as well, who tested the 
programs with their classes or served as controls and advised on design and revision. 

Project Description 

1993- 1994 

We assembled a group of faculty-poets and teachers of poetry— to advise us. We worked 
out questions for and programmed a preliminary version of "Design" and tested it in three classes. 
We decided that "Musee des Beaux Arts" would be our second program, and began to collect 
material on it. Meanwhile, based on the students' answers in the preliminary version of "Design" 
we revised the questions and programmed the computer's responses to their answers,in a more 
complete version of the program. The procedure that we developed to do this-the conversion of 
the guided discussion of the poems into interpretive categories, on which the programming design 
can be based— constitutes one of the major outcomes of the project. In the Spring we tested 
"Design" in four classes and compared the results with two control classes. A summer school 
class at Kingsborough used a revised version of "Design. " Preliminary programming was done on 
"Musee." 

1994- 1995 

We finished the preliminary version of "Musee des Beaux Arts" and tested it in the Fall 
semester with six different classes, five at Kingsborough and one at New Jersey Institute of 
Technology. Two of the courses used "Design" as well. Since this was the preliminary version, 
we did not go through the same testing procedure as we had in the spring, but on the basis of 
students' answers and their suggestions, as well as the suggestions of the teachers, we revised the 
questions, and finally redesigned the entire format of the program. We discovered that, since each 
poem is unique, each demands its own approach to program style and programming technique. 
Accordingly, we developed an innovative method for using faculty and student input to determine 
programing design. 

During the fall semester, we exhibited our work to date at a poster session at the FIPSE 
Project Directors' Meeting , did an hour-long presentation at the Annual Conference of the 
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League for Innovation in the Community College in Houston, and demonstrated our work at 
"Learning Technologies," a CUNY Faculty Senate conference. 

In the Spring we tested the revised "Musee des Beaux Arts" in three Kingsborough 
classes. We also worked with our faculty group to revise the database for "Design" and to work 
out questions for "The Windhover." Because this is such a complex and multi-layered poem, 
making up even the preliminary version of the questions was difficult. We began work on a 
program dealing with prosody and the sonnet form, called "Poetry to the Ear." In May, we 
sponsored a day-long conference at our College, "Computers and the Humanities," with 
presentations by a number of academic software designers within CUNY and in other colleges in 
the region, including ourselves. 

During the summer, after further revision, we tested "Musee" with students again and 
worked with our outside evaluator, Mr. Mohamed Tazari. We traveled to the University of 
Virginia to visit the IATH (Institute for Advanced Technology in the Humanities), and in August, 
Project Director Nimchinsky attended the AACE (Association for the Advancement of 
Computing in Education) World Conference on Artificial Intelligence in Education in Washington, 
DC. Programming of the preliminary versions of "Poetry to the Ear" and "The Windhover" 
continued throughout the summer. 

Evaluation/ Project Results 

We used several methods of evaluation throughout the project: we compared the work of 
students who used the programs with that of control groups; we asked students to fill out a 
questionnaire after each use of the software, including their suggestions for improvement; we 
included files at the end of the programs which gave us valuable records of how the programs 
were used; we consulted often with our faculty group; we consulted an outside evaluator. 
Feedback from students and faculty was overwhelmingly positive and very useful in that we were 
able to implement many of their suggestions. Statistical results were inconclusive, but seemed to 
indicate that those who benefited most from the programs were able to apply what they had 
learned to the study of another related poem. The two-year time limit of the grant meant that it 
was impossible to both create the programs and test them with large enough groups to obtain 
meaningful statistical results. 

We are continuing the project. At present, we are testing the preliminary "Windhover" 
and creating the preliminary version of "Poetry to the Ear." We do intend to complete both of 
these programs and to test them with students at Kingsborough and at other campuses whenever 
possible. We continue to test and revise the earlier programs. In addition, we hope to extend the 
current project next year by creating a CD-ROM on several poems of the Romantic period, with 
the help of scholars in the field. We are currently seeking funding for this endeavor. In the 
meantime, we are ready to approach publishers with the first three programs. 

Summary and Conclusions 

Despite the fact that each program took longer to develop and test than we had expected, 
and that rapidly changing technology meant that we had to constantly rethink and redesign, we 
believe our project was successful. We completed fewer programs than we had projected, but the 
software we did design was much more sophisticated and complex and contained many more 
features than we had originally planned. 
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Exploring Poetry Through Interactive Computer Programs 

Kingsborough Community College of the City University of New York 
2001 Oriental Boulevard 
Brooklyn, NY 11235 

Project Directors: Howard Nimchinsky and Jocelyn Camp 
(718) 368-5939 

Final Report 

Project Overview 

The purpose of our project was to design, test, and evaluate several distinct software 
programs, each on an individual poem. Each program would allow students in introductory 
literature and poetry courses to explore the poem in depth and develop an interpretation of it. 
With the help of a group of faculty, with whom we consulted throughout the project, we chose 
the poems to be used, and in the first year of the grant tested preliminary and revised versions of 
the first program, on Robert Frost's "Design," and completed the preliminary version of a second 
one, on W.H. Auden's "Musee des Beaux Arts." In the second year, we presented our work at 
three conferences, tested and revised "Musee des Beaux Arts" extensively, continued planning 
and began the programming for two more programs (one on Gerard Manley Hopkins' "The 
Windhover," and one on elementary prosody, called "Poetry to the Ear") and hosted a conference 
on "Computers and the Humanities" at our college. Students at a community college and a senior 
college used the programs during the two years of the grant in a total of eighteen poetry or 
introductory literature courses, and several other classes served as control groups. Many 
suggestions made by these students and teachers were implemented in later versions of the 
programs, as well as those of an outside evaluator on educational software design. 
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Purpose 

The problem our project addressed was the difficulty that students in introductory 
literature classes often have with close reading of poetry, and their reluctance to do so. Our goal 
was to emphasize close reading and analysis. During the course of the grant, our understanding 
of the problem itself did not fundamentally change. In fact, the ways the students used the 
programs made us aware of reading difficulties we had not anticipated. The records of each use 
of the programs indicate how students go about reading and analyzing; this taught us more about 
their learning processes than we could have discovered through traditional classroom instruction 
(see "Evaluation" below for a description of these records.) The greatest problem many of them 
have, we found, was in putting together the pieces of information they had learned into a coherent 
overall interpretation. According to the data we collected, the programs were for the most part 
successful in focusing attention on the language of the poems and allowing students to do very 
detailed close reading. The specific questions they were asked to answer encouraged this, and the 
nature of the computer itself commands concentration. Certainly most students enjoyed using the 
programs. They particularly liked the ease with which they could access information about 
individual words and concepts. They could work at their own speed, and this varied greatly from 
student to student. 

Any administrative pitfalls involved in using our software would have to do with providing 
the requisite technology to support it. Technology is developing and changing at such a rapid 
pace that both the software designer and the campus must constantly be prepared to adjust. The 
college must be responsible for keeping hardware up to date, an expensive proposition in days of 
tightened budgets. College technicians must be on hand for the multitudinous things that can go 
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wrong with the equipment. Teachers (especially those who are unaccustomed to using computers 
with their classes) need some instruction in using the software, though the programs are really 
self-explanatory and "computer literacy" is not required. 

Those who embark on software development themselves will learn, as we did, that 
everything takes longer than expected; equipment breaks down, unforeseen bugs occur, viruses 
appear. Constant adjustments have to be made as one perceives how the programs are actually 
used. For example, we discovered that each poem requires an entirely different approach and that 
we could not simply copy the procedures we had already worked out. New developments in 
technology allow for more and more sophisticated design and more and more possibilities, but 
with each new invention adjustments in the plans have to be made. These are exciting times for 
software development, but the speed of change in the field requires that expectations must 
undergo constant revision. 

An exciting outcome of our project was the unanticipated development of a method of 
programming complex relationships through the input of faculty participants. Originally, we 
ourselves determined the possible interpretations of a poem and decided how the students' 
understanding of individual parts of the poem might fit into those interpretations. We found that 
the program based on our judgments was to some extent faulty, chiefly in the appropriateness of 
some of the comments the program made on the students' answers. When we met with our 
faculty participants, we realized that they brought to the poem in question a wider range of 
understanding than we had originally conceived. We assigned them the task of developing a list 
of possible interpretations of the poem as a whole. They discussed the lists submitted by each of 
them at the meetings we held regularly, and, under Dr. Camp's guidance, they ultimately reached a 
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consensus. We then designed a grid which the group would use to indicate what readings of parts 
of the poem were to be judged consistent or inconsistent with the various interpretations. (Please 
see Appendix E.) 

After we administered the preliminary version of the program to the classes using it, we 
asked the group to categorize the answers that students gave to specific questions, each expressed 
by the students in their own way. Once the categories were formulated, we distributed the grid to 
the participating faculty. On it, each teacher was to indicate the relationship between a category 
of answer and the interpretations agreed on earlier. Once the grids were completed, discussed by 
the group at large, and merged. Dr. Nimchinsky was able to use the resulting master grid to create 
the database of responses and to modify the programming design. This method made it possible 
for the program to respond to the answers which the students give in their own words and to 
judge the internal consistency of their replies. 

What we have described may seem a well thought out and organized approach to creating 
a program dealing with text, and that is indeed what it turned out to be. In its development, 
however, it evolved in stages that we at first did not notice or appreciate. Eventually, it became 
clear that we had hit upon an extremely effective methodology for dealing with textual analysis 
and interpretation. Moreover, we learned from this process that there are no short-cuts to 
developing this type of software, that neither the programmer's or the scholar's ivory tower 
provides a good vantage point for dealing with the reactions of others— especially students. W e 
feel at this point that, in the future, we can 1) teach our approach to other teachers and 
programmers, and 2) extend this methodology to other fields in which the interpretation of texts is 
crucial. In the meantime, we are eager to apply our method to other poetic works. 
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Background and Origins 

At Kingsborough, all English faculty teach developmental as well as literature courses. 

Our FEPSE project originally developed out of our work as reading teachers. In the late 1980's 
we designed a developmental reading exercise, programmed in Prolog, in which students would 
answer questions on a short passage of text and the computer would respond. With the help of a 
series of small grants we wrote five of these programs and used them in reading courses. We 
realized that the same basic techniques could be applied to literature, and determined to create 
several programs in the interpretation of poetry. 

Support from the College has been essential to our work on the FIPSE grant. One 
necessity for the success of a project like ours is the cooperation of the academic computing staff 
of the college, which, for the most part, we had. Since we used the software in the College's 
computer labs, their help was essential in setting up and supervising labs. They also gave us 
access to equipment (such as scanners), and allowed us to put updated versions of our programs 
on the network. The College provided computer and printer upgrades for the Project Directors' 
offices, access to the Toolbook hotline, and some travel funds. We were dependent upon our 
cooperating faculty from Kingsborough and other campuses as well, who tested the programs 
with their classes or served as controls and advised on design and revision. 

Project Description 

The format of the software, programmed in Toolbook and Prolog, is as follows: students 
are presented with a copy of the poem on screen, and may access information about the author, 
the date of publication, definitions of the words in the poem, and other background information 
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by clicking on "hotwords." They may also hear it read aloud and, in the case of "The Windhover," 
see a video of the bird of the title in flight. When they are ready, they write in answers to a series 
of questions on the poem aimed at preparing them to write an overall interpretation. Questions 
are not judged right or wrong, but in "Design" the program "reads" their answers and asks if it has 
read them accurately. If the program cannot understand an answer, it offers a number of 
possibilities and asks students which they agree with and which they do not. The program also 
keeps track of the answers and checks the consistency of the interpretation, as measured against 
several different ways of reading the poem. In "Musee des Beaux Arts," there is quite a bit more 
background information, as the purpose of the program is to teach the importance of 
understanding allusions in poetry. Students can see the paintings referred to in the poem and read 
the story of Daedalus and Icarus. They write a preliminary interpretation, before they study the 
background information, and can compare it side by side with the interpretation they write at the 
end. Unlike "Design," which is a poem that can be read several different ways, depending upon 
how one interprets the images in the poem, "Musee" is relatively unambiguous; hence the 
consistency check is unnecessary. "The Windhover," which will eventually be the most complex 
of the programs, contains features of both. It is in the process of initial testing with students; the 
computer's replies to their answers cannot be programmed until we discover what they actually 
will write. "Poetry to the Ear" is a series of interactive exercises and is completely different from 
the other programs, but will be used as a supplement to them. 

We include several files at the end of the programs which provide information about how 
the programs were used by the students. These are described below, on page 9. Additional files 
enable the program to return the students to the program for additional sessions preserving all the 
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conditions which obtained at the time they last left the program. This "bookmark" feature is 
indispensable when the students are using the program on a local area network, since the program 
itself cannot be modified. Likewise, at such time as we may include these programs on a 
CD-ROM, it will be impossible to create a bookmark, except through external files. 

Project Activities 
1993-1994 

As the grant began we assembled a group of faculty-poets and teachers of poetry-to 
advise us. We worked out questions for and programmed a preliminary version of "Design" and 
tested it in three classes. The faculty group met to discuss poems to be chosen for the next 
program. We decided on "Musee des Beaux Arts," and began to collect material on it. 
Meanwhile, based on the students' answers in the preliminary version of "Design" we revised the 
questions and programmed the computer's responses to their answers, in a more complete version 
of the program. 

In the Spring semester we tested "Design" in four classes (three at Kingsborough and one 
at New Jersey Institute of Technology) and compared the results with two control classes (one at 
Kingsborough and one at Brooklyn College). (Please see Evaluation/Project Results below for a 
description of the testing process and results.) A summer school class at Kingsborough used a 
revised version of "Design." Preliminary programming was done on "Musee." 
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1994-1995 

We finished the preliminary version of "Musee des Beaux Arts" and tested it in the Fall 
semester with six different classes, five at Kingsborough and one at New Jersey Institute of 
Technology. Two of the courses used "Design" as well. Since this was the preliminary version, 
we did not go through the same testing procedure as we had in the spring, but on the basis of 
students' answers and their suggestions, as well as the suggestions of the teachers, we revised the 
questions, eliminating the ones that didn't work, and finally redesigned the entire format of the 
program, making it possible for users to access the questions from the main screen, answer the 
questions in any order, and revise any answer at any time. We discovered that, since each poem is 
unique, each demands its own approach to program style and programming technique. 

Also during the fall semester, we exhibited our work to date at a poster session at the 
FIPSE Project Directors' Meeting , and did an hour-long presentation at the Annual Conference 
of the League for Innovation in the Community College in Houston. FIPSE grantee Dean 
Savage, who had seen our poster session at the FIPSE meeting, invited us to demonstrate our 
work at "Learning Technologies," a CUNY Faculty Senate conference held in December. 

In the Spring we tested the revised "Musee des Beaux Arts" with three Kingsborough 
classes, two of which were Introduction to Literature courses and one of which was an 
Introduction to Poetry course. (See Evaluation / Project Results) Our Program Officer, Brian 
Lekander visited and observed Dr. Camp's poetry class. As luck would have it, all the students' 
disks had been infected by a virus and they were unable to use the program that day. (They wrote 
their first interpretations on paper while Brian looked at the program on a separate computer). 
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Also during the Spring semester we worked with our faculty group to revise the database 
for "Design" and to work out questions for "The Windhover." Because this is such a complex 
and multi-layered poem, making up even the preliminary version of the questions was difficult. 

We began designing a program dealing with prosody and the sonnet form. In May, we sponsored 
a day-long conference at our College, "Computers and the Humanities," with presentations by a 
number of academic software designers within CUNY and in other colleges in the region, 
including ourselves. Among the presenters were FIPSE grantees Paula Berggren and David 
Stephan. Bret Eynon of the "Who Built America" project was the keynote speaker. (Please see 
Appendix G for Conference program). We were visited in June by a group of Swedish foreign 
language professors from Malardalen University College who were touring some colleges and 
universities in the United States through the United States Information Agency. We showed them 
our college's computer facilities and demonstrated "Musee" and an ESL program developed by 
another Kingsborough faculty member (Cindy Greenberg-once a FIPSE grantee). 

During the summer, after further revision, we tested "Musee" with students again and 
worked with our outside evaluator, Mr. Mohamed Tazari. Mr. Tazari met with us several times 
and attended Dr. Nimchinsky's Introduction to Literature class twice to observe their use of the 
program. He made a number of suggestions on program design, most of which we were able to 
implement. (Please see below, Evaluation/Project Results, and see Appendix D for Mr. Tazari's 
report). 

We traveled to the University of Virginia to visit the IATH (Institute for Advanced 
Technology in the Humanities) in late June where we met with the Director, John Unsworth, 
Romantic scholar Jerome McGann, and the head of the Electronic Text Center, David Seaman. It 
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was a very productive trip. We learned at great deal and found the IATH to be a dynamic and 
exciting place. Especially enlightening was what we found out about the extensive electronic 
archives of literary and historical texts and graphics which they are busily assembling in several 
fields. We intend to make use of them and similar material collected at other universities in 
developing new programs. We were also made aware of the utility of the World-Wide Web as 

both a resource and a vehicle for our programs. 

In August, Project Director Nimchinsky attended the AACE (Association for the 
Advancement of Computing in Education) World Conference on Artificial Intelligence in 
Education, "AI-ED '95" in Washington, DC. At this conference, he learned about the use of 
Intelligent Tutoring Systems (ITS's) in the development of training programs. He saw that the 
technique was applicable to the kind of programs we are creating, especially those which deal 
with complex poetic texts. A logical extension of the approach we have used so far, the ITS 
teaches content and the skills needed to master that content at the same time. 

Programming of the preliminary versions of "Poetry to the Ear" and "The Windhover" 
continued throughout the summer. "The Windhover" design is different from that of the previous 
programs, in that there are many more questions, divided into categories. Users click on icons on 
the main screen to access questions in the four groups ("Events," "Ideas," "Imagery," and 
"Form"). We also obtained a video from a bird photographer of a kestrel flying, which we 
incorporated into the program. "Poetry to the Ear" is a series of interactive exercises. We are 
beginning with sonnet form, since both "Design" and "The Windhover" are sonnets. 
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Evaluation / Project Results 

We used several methods of evaluation throughout the project. 

1 . We asked students to fill out two kinds of questionnaires. One gave us 
information about students' previous English courses, what their reading habits were for 
poetry and in general, and how they perceived themselves as computer users. (Control 
groups answered the same questionnaire, minus the computer use question). Also, after 
each use of the software, we asked for their reactions, including what they liked best and 
least about the programs, their opinions of how clear the instructions were, information 
about how much they felt they had learned, and their suggestions for improvement. 

(Please see Appendix C.) The suggestions were extremely useful and we were able to 
incorporate many of them in revised versions of the programs. For example, students 
wanted to be able to go back and revise answers, and access questions directly from the 
main screen. We were able to allow for this in later versions. One of our faculty group 
suggested that students write "before and after" versions of their interpretations of 
"Musee," one before they had read the background material and one after so that we could 
compare results, and we added this feature. 

2. We tested students who used the programs and control groups. Three 
Kingsborough classes and one class at NJIT used "Design" in the spring of 1994. Two 
control classes were taught the poem without using the computer program, one at 
Kingsborough and one at Brooklyn College. All classes were asked to write on a 
follow-up poem with some similarities to "Design," Emily Dickinson's "Apparently with no 
surprise..." (The faculty group decided on this poem after long discussion and made up 
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the test.) Students were to make up study questions on the Dickinson poem before 
writing about it in class. The papers (the test on "Design," the study questions on 
"Apparently..." and the test on "Apparently...") were marked by several independent 
graders. Test results were inconclusive. Senior college students did better than 
community college students whether using the computer programs or not. However, 
those who used the computer program and did well on it were more likely to make up 
better questions on the Dickinson poem than those who did not use the programs. This 
seemed to indicate that those who benefited most from the programs were able to apply 
what they had learned to the study of another related poem. (Please see Appendix B for 
statistical results.) 

In the Spring 1995 semester, we tested "Musee des Beaux Arts." Three sections at 
Kingsborough (two in Intro, to Lit. and one in Poetry) used the program, and an Intro, to 
Lit. class at Kingsborough and a similar introductory literature class at LaGuardia 
Community College, CUNY served as our control groups. These students wrote a brief 
interpretation of the poem before studying it in class and another afterwards. Their papers 
were mixed with before-and-after interpretations written by students who used the 
programs and all were graded by a group of outside graders. Results, examined by our 
college statistician, showed that computer users' first essays were not as good as the ones 
in the control classes, but that in the end all students arrived at the same place. (Please see 
Appendix B.) 

Because we had made a number of changes in the program, we tested it again in 
the summer with an Introduction to Literature class and used another section of the course 
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as a control group. The results of this testing, graded in the same manner as were the 
papers in the spring, showed that the experimental group did slightly better than the 
control group. However, the sample was probably too small to prove very much. 

3 . We include several files at the end of the programs which give us valuable 
information about how the programs were used by the students. These give us insight into 
how they work with the software and also into their reading processes. One file keeps a 
chronological record of the students' sessions with the program. The time the students 
spend in each session is noted; each hotword which they consult, as well as the questions 
they attempt and their answers, are recorded in the order they followed. For instance, we 
can see how many times they go back to revise answers (along with all the versions of the 
answers) and understand, in many cases, reasons why the students adopt a particular 
strategy for exploring the program and the poem. Another file lists all the questions in 
numerical order and includes all the students' answers to a particular question immediately 
after the text of that question. Also included in this file are the interpretive essays which 
the program calls upon the students to write. This is the file which is printed out for the 
students, who thereby get an enhanced perspective on their work. (See Appendix F for 
the logs of one student's session with "Musee des Beaux Arts.") 

4. We asked an outside evaluator, Dr. Mohamed Tazari, to evaluate "Musee des 
Beaux Arts." Currently an Adjunct Professor of French at the College of Staten Island, 

Dr. Tazari is the developer of an interactive multimedia software program entitled "French 
Interactive." This was his dissertation project for his degree in Instructional Technology 
and Media in Education at Teachers College, Columbia University. He has also served as 
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Multimedia Coordinator at Teachers College, in charge of setting up and running a 
multimedia laboratory. He met with us in the summer of 1995, observed the students 
using the program, and made a number of suggestions for improvement. Most of these 
suggestions, having to do with making the program easier to use (the addition of a help 
screen, more efficient way of dismissing ’hotwords," etc.) we incorporated into "Musee" 
and will employ in future programs. (His report is included in Appendix D) 

5. In addition to the formal procedures described earlier in Purpose, we consulted 

often with our faculty group who were both advisors and testers of the programs with 
their classes. They created some of the questions for the programs, suggested design 
features, and offered numerous insights both on interpretation of the poems and pedagogy. 

Overall, feedback from students and faculty was overwhelmingly positive and very useful 
in that we were able to implement many of their suggestions. Statistical results, though favorable, 
were inconclusive. Although we tried to test in a way that would yield valuable data, we don t 
believe that the results we have so far are really significant. There are several reasons for this. 

One is that we had to learn how to set up the test. In the first year, we were not clear enough in 
our instructions to the teachers, and some of them used the test on "Design" as a final exam while 
others used it as a class exercise. More important was the fact that the programs were in the 
process of development while we were testing them. Student input was invaluable, but evaluating 
for statistical results was premature. We really needed a third year on the grant in order to do this 
properly. Essentially, we found that we were trying to perform too many different kinds of tasks 
at the same time, especially since we did not use a separate computer programmer and all actual 
programming was done by Project Director Nimchinsky. 
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We are continuing the project. At present, we are testing the preliminary "Windhover" 
and creating the preliminary version of "Poetry to the Ear." We do intend to complete both of 
these programs and to test them with students at Kingsborough and at other campuses whenever 
possible. We continue to test and revise the earlier programs. In addition, we hope to extend the 
current project next year by creating a CD-ROM on several poems of the Romantic period, with 
the help of scholars in the field. We are currently seeking funding for this endeavor. In the 
meantime, we are ready to approach publishers with the first three programs. 

Summary and Conclusions 

Despite the fact that each program took longer to develop and test than we had expected, 
and that rapidly changing technology meant that we had to constantly rethink and redesign, we 
believe our project was successful. We completed fewer programs than we had projected, and 
our aims were revised somewhat. For example, we had initially planned to make the programs 
available for Apple as well as IBM computers. However, since advances in technology now make 
it possible to convert form one to the other, we felt our time was best spent continuing to work 
within the IBM platform. The software we did design was much more sophisticated and complex 
and contained many more features than we had originally planned. We believe that the project 
demonstrated the effectiveness and potential for such software. 



O 
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Appendix A 

Information for FIPSE 

One of the things we liked best about the way FIPSE operates is that you were always accessible 
to us and willing to help. Both Brain Lekander and Dora Marcus gave us advice and direction 
and answers to our numerous questions throughout the period of our grant. At the same time, we 
felt that FIPSE believed in what we were doing and that we were trusted to proceed as we best 
saw fit. In other words, no one was constantly looking over our shoulders, and yet, your 
oversight was uniformly helpful, and assistance and encouragement were there when we needed 
them. 

Our only suggestion has to do with the Project Directors' Meeting. We enjoyed both conferences 
we attended very much. We learned a great deal, both about the work being done on other grants, 
and also on how other grantees had proceeded to implement and evaluate their work. We would 
have liked to have gone this year too! Our suggestion is that there might be some way to include 
ex-grantees in the Project Directors' Meeting~and/or perhaps they could meet as a separate group 
to discuss life after FIPSE. As we determine where to go with our work now that the grant is 
finished, the advice and suggestions of other people at the same stage would be very helpful. 

We definitely encourage FIPSE to fund other software development projects in the Humanities. 
Intelligent programs are just now beginning to take advantage of the technological possibilities 
that are now available. At this point, colleges and universities are still unsure if such work merits 
serious consideration as "scholarship" and tend not to include it in tenure and promotion 
considerations. Therefore, faculty who already have tenure and rank (for example, the two of us, 
Stuart Curran at Penn., Jerome McGann at UVa) are the only ones who dare pursue it. This 
attitude will probably change, especially since there is such an educational necessity for such 
projects. In the meantime, encouragement and funding by FIPSE can be a major factor in 
altering the landscape of academia to include opportunities and rewards for innovations in 
education using new technologies. 
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Appendix B 

Statistical Results of Testing 



In the following pages, the reports on the testing of Spring 1994, Spring 1995, and Summer 1995 
precede the data concerning student profiles. 

In the test report for Spring 1994, Score 1 designates the essay test on the poem "Design." Score 
2 refers to an essay test on a related poem. Score Q is the evaluation of the students' attempts to 
generate questions on the latter poem. 

In Spring 1995, the designations Prel, Pre2, Postl, and Post2 refer to the scores assigned by first 
and second readers to the interpretations which the students wrote about the Auden poem before 
and after studying the poem, the experimental group through the program and the control with 
their teachers. 



Spring Semester /??y 







Score 1 


Score 2 


Score Q 


N 


Mean s.d. 


Mean s.d. 


Mean 


s.d. 


Experimental 


98 


8.88 5.57 


10.53 6.38 


8.32 


7.55 


Control 


56 


10.77 5.63 


9.96 7.01 


5.77 


6.18 


t ( df=1 52) 




2.01 


0.51 


2.16 




2 




p < .05 


p > .05 


p < .05 






r> r> 

4 0 



CTR0EXP1 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 




0 


42 


44.7 


44.7 


44.7 




1 


52 


55.3 


55.3 


100.0 




Total 


94 


100.0 


100.0 





Valid cases 



94 



Missing cases 



Number of 


valid observations 


(listwise) = 


94 


.00 












Valid 


Variable 


Mean 


Std Dev 


Minimum 


Maximum 


N 


PRE1 


1.60 


.81 


1 


4 


94 


PRE2 


1.71 


.82 


1 


4 


94 


POST1 


2.65 


.95 


1 


4 


94 


POST2 


2.68 


.93 


1 


4 


94 



Number of valid observations (listwise) 



94.00 



Variable 

PRETOT 

POSTOT 



Mean 


Std Dev 


Minimum 


Maximum 


Valid 

N Label 


3.31 


1.49 


2.00 


8.00 


94 


5.33 


1.72 


2.00 


8.00 


94 



GAIN 











Valid 


Cum 


Value Label 


Value Frequency 


Percent 


Percent 


Percent 




-2.00 


1 


1.1 


1.1 


1.1 




-1.00 


4 


4.3 


4.3 


5.3 




.00 


16 


17.0 


17.0 


22.3 




1.00 


13 


13.8 


13.8 


36.2 




2.00 


22 


23.4 


23.4 


59.6 




3.00 


20 


21.3 


21.3 


80.9 




4.00 


12 


12.8 


12.8 


93.6 




5.00 


5 


5.3 


5.3 


98.9 




6.00 


1 


1.1 


1.1 


100.0 




Total 


94 


100.0 


100.0 




Valid cases 94 


Missing cases 


0 








t- tests for independent 


samples of CTR0EXP1 










Number 










Variable 


of Cases 


Mean 


SD SE 


of Mean 


GAIN 












CTR0EXP1 0 


42 


1.3571 


1 . 


527 


.236 


CTR0EXP1 1 


52 


2.5577 


1 . 


589 


.220 




Mean Difference = -1.2005 



Levene's Test for Equality of Variances: F= .001 P= .978 




< ' 



t-test for Equality of Means 
Variances t-value df 2-Tail Sig 



SE of Diff 



95% 

Cl for Diff 



Equal -3.70 92 .000 .324 

Unequal -3.72 89.22 .000 .323 



(-1.844, -.557) 
(-1.842, -.559) 



- - Description of Subpopulations 

Summaries of PRETOT 

By levels of CTR0EXP1 



Variable 


Value Label 




Mean 


Std Dev 


Cases 


For Entire Population 




3.3085 


1.4885 


94 


CTR0EXP1 


0 




4.0000 


1.6965 


42 


CTR0EXP1 


1 




2.7500 


1.0073 


52 


Total Cases 


= 94 












- - Description 


of 


Subpopulations - - 






Summaries of 


POSTOT 










By levels of 


CTR0EXP1 










Variable 


Value Label 




Mean 


Std Dev 


Cases 


For Entire Population 




5.3298 


1.7189 


94 


CTR0EXP1 


0 




5.3571 


1.6795 


42 


CTR0EXP1 


1 




5.3077 


1.7661 


52 


Total Cases 


= 94 












- - Description 


of 


Subpopulations - - 






Summaries of 


GAIN 










By levels of 


CTR0EXP1 










Variable 


Value Label 




Mean 


Std Dev 


Cases 


For Entire Population 




2.0213 


1.6655 


94 


CTR0EXP1 


0 




1.3571 


1.5273 


42 


CTR0EXP1 


1 




2.5577 


1.5893 


52 



Total Cases = 94 







30 



Summer Semester /f?£" 



Prescore Total Postscore Total Gain 





N 


Mean 


s.d. 


Mean 


s.d. 


Mean 


s.d. 


Experimental 


22 


1.63 


0.62 


2.59 


0.73 


0.96 


0.84 


Control 


24 


1.60 


0.57 


2.44 


0.77 


0.84 


0.55 



t (df = 44) 0.58 

n P > .05 
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Means for Different Levels of 
Experience with Computers, Remediation, and 
Having Literature Courses 



Means 







Score 


1 


Score 2 




Score Q 






N 


Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


Experience With Computer 














A 


2 


8.00 


4.24 


4.5 


6.36 


8.00 


11.31 


B 


27 


10.85 


5.57 


12.11 


6.07 


10.19 


7.15 


C 


23 


9.78 


5.88 


12.39 


5.69 


10.22 


7.59 


D 


18 


9.11 


5.14 


12.22 


5.31 


10.06 


7.00 


E 


49 


10.82 


5.49 


10.74 


6.30 


6.08 


6.43 


Analysis of Variance 


d.f. 


F 


P 


F 


P 


F 


P 




1,114 


0.52 


>.05 (NS) 


1. 14 >.05 


(NS) 


2.44 


=.0508 



Remediation 



0 




3 


11.67 


4.04 


16.33 


2.08 


16.67 


0.58 


i 




62 


11.71 


5.62 


11.83 


6.32 


7.98 


7.48 


2 




54 


8.64 


4.98 


10.83 


5.70 


8.52 


6.71 


Analysis of Variance 


d.f. 




F 


P 


F 


P 


F 


P 




2,116 




4.89 


<.01 


1.40 


>.05 (NS) 


2.16 


>.05 (NS) 



Having Literature Courses 



0 


92 


9.52 


5.75 


9.43 


6.75 


5.78 


6.89 


i 


48 


9.67 


5.00 


11.33 


6.15 


9.98 


6.54 


2 


11 


8.00 


7.38 


13.18 


5.86 


10.55 


8.51 


3 


2 


15.00 


5.66 


8.00 


11.31 


0.00 


0.00 


4 


1 


15.00 




17.00 




12.00 




Analysis of Variance 


d.f. 


F 


P 


F 


P 


F 


P 




4,149 


0.91 


>.05 (NS) 


1.55 


>.05 (NS) 


4.21 


<.01 


Total 


154* 


9.56 


5.65 


10.32 


6.60 


7.40 


7.17 



*AII variables are not present for every case 





Means for Different Levels of Experience with Computers 
by Experimental and Control Groups 





Score 1 


Score 2 


Score Q 


Cell Sizes 


Experience with Computer 


Experimental Control 


Experimental Control 


Experimental 


Control 


Experimental Control 


Expert 


8.67 10.96 


9.67 10.80 


6.33 


6.07 


3 46 


Experienced 


9.11 


12.22 


10.06 


.. 


18 0 


Fairly Capable 


9.78 


12.39 


10.22 


.. 


23 0 


Beginner 


10.85 


12.11 


10.19 


.. 


27 0 


Not Interested 


8.00 


4.50 


8.00 


•• 


2 0 


Analysis of Variance 


F Sig of F 


F Sig of F 


F 


Sig of F 




Experience with Computer 


0.52 0.72 


11.14 0.34 


2.42 


0.05 




Experimental Group 


0.48 0.49 


0.10 0.75 


0.01 


0.95 




Total Explained by Model 


0.51 0.77 


0.93 0.47 


0.19 


0.94 






o q 







Experimental Control 





/iff 




» Experimental -®- Control 




<> r- 

<30 



'ffV 




■ Experimental -®- Control 




O p> 

oo 






Means for Different Levels of Remedial Coursework 
by Experimental and Control Groups 





Score 1 


Score 2 


Score 


Q 


Cell Sizes 






Experimental Control 


Experimental Control 


Experimental 


Control 


Experimental Control 


Remedial Courses 














0 


11.67 


16.33 


16.67 


.. 


3 


0 


1 


10.78 13.00 


11.78 11.88 


12.08 


2.31 


36 


26 


2 


8.85 8.30 


11.68 9.40 


7.09 


10.95 


34 


20 


Analysis of Variance 


F Sig of F 


F Sig of F 


F 


Sig of F 






Remedial Courses 


1.09 0.30 


0.97 0.33 


11.61 


0.01 






Experimental Group 


, 4.84 0.01 


1.28 0.28 


1.98 


0.14 






Interaction: Rem x Exp 


1.89 0.17 


1.09 0.30 


34.89 


0.01 






Total Explained by Model 


3.16 0.02 


1.56 0.34 


12.61 


0.01 











O if 



m 




Number of Remedial Courses 
Experimental ■*- Control 




33 







Number of Remedial Courses 
- Experimental -®- Control 
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Means for Different Levels of Literature Courses 
by Experimental and Control Groups 








Score 1 


Score 2 


Score Q 


Cell Sizes 






Experimental Control 


Experimental Control 


Experimental 


Control 


Experimental Control 


Literature Courses 














0 


8.32 12.00 


9.15 10.03 


7.44 


2.37 


62 


30 


1 


9.40 9.96 


12.96 9.57 


9.72 


10.26 


25 


23 


2 


9.25 4.67 


13.50 12.33 


12.50 


5.33 


8 


3 


3 


15.00 


8.00 


0.00 


.. 


2 


0 


4 


15.00 


17.00 


12.00 


•• 


1 


0 


Analysis of Variance 


F Sig of F 


F Sig of F 


F 


Sig of F 






Literature Courses 


4.19 0.04 


0.27 0.61 


5.30 


0.02 






Experimental Group 


1.12 0.35 


1.60 0.18 


5.28 


0.01 






Interaction: Exp x Lit 


2.90 0.06 


1.62 0.20 


3.10 


0.05 






Total Explained by Model 


2.06 0.05 


1 .42 0.20 


4.66 


0.01 












Number of Literature Courses 
* Experimental -®- Control 




42 



/ff* 





43 



r99'/ 
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"5 f/t 






Average Reader Ratings 





N 


Pretest 


Posttest 


Difference 


Experimental 


22 


1.64 


2.59 


0.95 


Control 


24 


1.60 


2.43 


0.83 





N 


Pretest 


Posttest 


Difference 


Had Remedial Courses 


25 


1.56 


2.36 


0.80 


No Remedial Courses 


18 


1.72 


2.78 


1.06 





N 


Pretest 


Posttest 


Difference 


Had Literature Courses 


3 


1.66 


2.59 


0.93 


No Literature Courses 


40 


1.63 


2.51 


0.88 



Readinq Outside of Class 


N 


Pretest 


Posttest 


Difference 


5 (A lot) 


9 


1.94 


2.67 


0.73 


4 


13 


1.58 


2.35 


0.77 


3 


14 


1.61 


2.71 


1.10 


2 


5 


1.20 


2.00 


0.80 


1 (None) 


0 






,, 



Readinq Poetry for Pleasure 


N 


Pretest 


Posttest 


Difference 


5 (A lot) 


0 


.. 




.. 


4 


1 


3.00 


3.00 


0.00 


3 


9 


1.72 


2.44 


0.72 


2 


20 


1.65 


2.68 


1.03 


1 (None) 


10 


1.30 


2.15 


0.85 



Computer Competence 


N 


Pretest 


Posttest 


Difference 


5 (Expert) 


1 


3.00 


3.00 


0.00 


4 


4 


1.25 


2.00 


0.75 


3 


5 


1.30 


2.70 


1.40 


2 


7 


1.93 


2.78 


0.85 


1 (Not introduced) 


0 


.. 


.. 


•• 
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Appendix C 
Student Questionnaires 



QUESTIONNAIRE: FIPSE Poetry Project 



Please assist us by answering the following questions to the best of your ability. If you need help 
with any item, please consult your instructor. 



Last Name . 



First Name . 



(i) 



( 2 ) 



Soc. Sec. No. * ■ Date of Birth 

( 3 ) 



( 4 ) 



College you are now attending . 



. Major . 



( 5) ( 6 ) 
How many college credits will you have at the end of this semester? _ 



( 7 ) 



Please list below the required English courses you have already taken. Include remedial courses, 
but not literature courses. 




Please list below all literature and advanced English courses you have taken. 
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QUESTIONNAIRE: FEPSE Poetry Project (continued) 



How much do you read for pleasure on a regular basis? ( 23 ) 

A great deal every day A moderate amount every day — 

(a) (h) 

A moderate amount several times a week Very little — Not at all — 

(c) (d) W 

How much poetry do you read for pleasure on a regular basis? (24) 

A great deal every day A moderate amount every day — 

(a) (b) 

A moderate amount several times a week Very little — Not at all 

(c) (d) to 

Do you spend time on creative writing on a regular basis? Yes — No — 



If you answered 'yes,' indicate below what you have written. Check all that apply. (27) 
Short Story Novel _ Poetry _ Play_ Literary Criticism _ Essay 

(a) (b) (C) (d) < e > (0 



Other Please Describe: 

(g) 



(h) 



QUESTIONNAIRE: FEPSE Poetry Project 



Please assist us by answering the following questions to the best of your ability. If you need help 
with any item, please consult your instructor. 



Last Name 



First Name 



(i) 



( 2 ) 



Soc. Sec. No. - - Date of Birth 

O) 



(4) 



College you are now attending . 



. Major 



(5) (6) 

How many college credits will you have at the end of this semester? _ 



(7) 



Please list below the required English courses you have already taken. Include remedial courses, 
but not literature courses. 




Please list below all literature and advanced English courses you have taken. 




(15-21) 



(Continued on the other side.) 



QUESTIONNAIRE: FIPSE Poetry Project (continued) 



How do you regard yourself as a computer user? (22) 

Expert Experienced Fairly Capable — Beginner — Not Interested 

F W - (b) <«> (d) (e) 

How much do you read for pleasure on a regular basis? ( 23 ) 

A great deal every day A moderate amount every day — 

(a) (b) 

A moderate amount several times a week Very little — Not at all 

(c) (d) (0 

How much poetry do you read for pleasure on a regular basis? ( 24 ) 

A great deal even/ day A moderate amount every day — 

5 (a) M 

A moderate amount several times a week _ Very little _ Not at all 

(c) (d) (e) 

Do you spend time on creative writing on a regular basis? Yes — No 



If you answered 'yes,' indicate below what you have written? Check all that apply. (2?) 
Short Story Novel _ Poetry _ Play_ Literary Criticism _ Essay 

(If (b) (c) (d) <•> (f) 



Other Please Describe: 

(g) 



(h) 



Poetry Project Follow-up 



Date 



You can be a great help to us as we work on improving our poetry software by offering your comments, 
suggestions, and criticism. Please take a few minutes to answer the following questions: 

1. Were the instructions within the program clear? If there were problems, please describe them. 



2 . Do you feel that you learned 

(a) a great deal about the poem 

(b) some information about the poem 

(c) nothing about the poem 

3. Do you feel that you learned 

(a) some information about poetry in general 

(b) a little bit about poetry 

(c) no information about poetry in general 

4. What did you like best about the program? 



5. What suggestions for improvement would you make? More information? Less information? Fewer questions: 
More questions? Different kinds of questions? (Explain) Different kinds of graphics? Different instructions? 



6. If you have used both "Design" and "Musee des Beaux Arts," did you prefer one to the other? If so, which one 
did you prefer and why? \ 
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Appendix D 

Report by Dr. Mohamed Tazari 



t * V*W) ills.- 



Formative Evaluation of the Kingsborough Multimedia Poetry 

Project 

by 

Dr. Mohamed Tazari 

The objective of this report is to document the review 
process of the poetry software application conducted during 
the summer session of 1995 as part of the formative 
evaluation process. The poetry software application was used 
in conjunction with the English Literature course at KBCC. 
Phase I consisted of a review of the application with the 
design team. During this session the main features and 
characteristics of the application were discussed including 
the intended objectives. After reviewing the application in 
light of its educational objectives, few recommendations 
were proposed to insure comprehensibility and flexibility of 
the interface. Phase II on the other hand consisted of 
observing students using the application in its intended 
instructional setting. A particular focus was on observing 
and documenting the interactive process between the learners 
and the multimedia instructional tool. Problems closely 
related to comprehensibility and flexibility of and access 
to the tools were identified, recorded and analyzed. 

Feedback from the students elaborating on such problems 
constituted the basis for proposing and recommending 
fundamental modification and reshaping of certain 
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instructional features and technical design elements so that 
the application meets the learning needs of the target 
audience . 



Phase I: Preliminary Review of The Application and 

Discussion with The Design Team 

In light of the discussion on the educational 
objectives of the instructional tools of the application 
with the design team and after conducting a preliminary 
review, the following recommendations were proposed: 

The help element of the instructional tool which 
explains the use and the objective of the various features 
should bo made available at all times from within the main 
instructional menu. 

In the version of the software that was initially 
reviewed during this phase, the help feature was available 
only at the opening menu. Reading and learning about the use 
and the capabilities of the various features of the 
application before seeing the layout of the main 
instructional menu and experimenting with the capabilities 
of the features can be very confusing for the learners for 



the following reasons: 



3 



1) learners have no visual representation of what they 
are reading 

2) by the time they reach the main menu they have 
already forgotten what they have read 

3) once in the main instructional menu they can not 
access the help feature as and when they need it 

This recommendation was integrated in the newer version 
of the application used by the students . This design 
modification provided students with the ability to 

request help about the various features as they engaged with 
the instructional tool. 

The Gallery component should be accessed not only 
independently as it is the case in the current version but 
also from within the poem 

The Gallery component of the application contains a 
collection of images which inspired the author of the poem. 
The purpose of this component is to provide students with 
access to the art work that inspired the poet and influenced 
both the content and form of the poem under examination. 

This feature allows the students to appreciate the visual 
context that influenced the ideas of the poem. 

The rational is that the contextual links and access to 
the Gallery from within the poem should be provided so that 
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the students can access not only the ' primary sources 
influencing the creative process in writing the poem but 
also they should be able to refer to these sources in 
context while interacting with the poem to gain a deeper 
appreciation of its meaning and capitalize on the benefits 
of the hypermedia capabilities of the instructional tools. 

Phase lit Observation of the Use of the Application by the 
Students At the Instructional Laboratory 

The use of the application by the students at the 
instructional laboratory was observed during two consecutive 
days. Students used the application to perform the following 
tasks: 1) acclimate themselves with the instructional tool, 

2) experiment with and use the various interactive features 
to conduct on-line research on and analysis of the poem and 

3) on-line composition and editing of responses to the 
various essay questions. 

At the beginning of each session, each student was 
given his or her own individual diskette to store the on- 
line responses to the various essay questions. Directions on 
how to access both the network and the software application 
were written on the board. After accessing the application 
students were instructed to explore its many features and 
respond to as many essay questions as they can and / or 
desire. 
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While students were engaged with .the application, this 
researcher observed individual or group of students 
interacting with it without ever interfering with the 
learning process and took notes of the various design 
features and elements that the students seemed to find 
confusing in terms of comprehensibility, flexibility and 
ease of use of the instructional tool. Whenever students 
seemed to be experiencing a problem, the nature of such a 
problem was identified, recorded and analyzed. At the end of 
each session, students were asked to reflect on their 
learning experience with the application and express in 
their own words what they liked about the application as 
well as the type of problems and difficulties they 
encountered while using the application. 

The analysis of the data sources composed of 1) this 
researcher’s observations and 2) students' reactions to 
their learning experience with the application revealed the 
following findings which were reported to the design team 
after each session and informed the recommendations for 
improvement which were proposed. 

The following design elements were identified and were 
categorized as being confusing for the learner since they 
require additional time and effort on tasks that do not 
support the learning process and could even inhibit or 
prevent learning. 




c: ►** 

o) ( 



BEST COPY AVAILABLE 



6 



Studdot s had to drsff tho iqousq in all directions over 
the text to locate parts of the text that contain additional 
information. 

Recommendation: Any part of the text that contains 

additional information should be highlighted in a specific 
color. 

Action* The highlighting feature was integrated in the 

last version of the application. 

Outcome t Students did not waste time looking for and 
finding parts of the text with additional information. 
Instead, they used that time to remain focused on 
supporting their comprehension needs by immediatly accessing 
and consulting additional information if it exits when and 
if needed. 

When etudonts clicked on any part of the text containing 
additional information, it was not clear how to close the 
field containing the new information. Only clicking on Poem 
would close the opened field. 

Recommendation: Clicking outside the opened field would 
immediatly close the field. 
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Actiont This modification was integrated in the last 
version of the application 

Outcome > Students did waste time trying to close the 
opened field to carry on their activities. Instead, they 
used that time to remain focused on the task at hand and 
process the information they have just read or consulted. 

While in the process of composing or editing responses, 
students had to save and close their work first before they 
are able to 1) verify or confirm their understanding and 2) 
s up port their writing process by consulting the poem or 
additional information. 

Rec omme ndation! Students should be able to continue 
their research even while in the process of writing. Indeed, 
they should be able to write, read and consult various types 
of documents simultaneously. 

Action: This recommendation requires more time to 
implement . 

Outcomes Students will not forget why they wanted to 
read or consult other types of documents in the first place 
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and will avoid ending up engaging in another type of 
research activity not closely related to the one they were 

working on. 

The software review conducted by this researcher as part 
of the formative evaluation process isolated only those 
design elements that would have prevented the application 
from achieving its intended educational objectives had they 
not been identified. The reason why each design element was 
not accomplishing the objectives sought were discussed and 
solutions were provided. The integration of the 
recommendations outlined throughout this report would 
provide a more appealing, comprehensible, accessible and 
responsive interface and would enhance the effectiveness of 
the learning experience. 
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Appendix E 

Grid used for Programming 

Interpretations of "Design" are followed by the grid used to categorize answers 
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Overall Interpretat ions of “Design 



1 . 



The poet is not sure if the scene was designed or random. 



2. The scene is ugly because it is a scene of death, but still 
there is good (nature operating as it must) and beauty (the white 
des i gn ) in it. 



3. However the scene appears, that's the way nature works. 



4. An evil power designed this scene of sickness and death. 

5. Frost creates his own design (the poem) through his descrip- 
tion of the scene and this is what the poem is basically about. 



6. If there is no designer, the situation is worse than if the 

designer is evil: the world operates at random. 

7. The poet views the natural world objectively and dis- 
passionately and does not offer an interpretation of the scene. 
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Appendix F 

Print-outs of Student Sessions 
Logs of one student's work on "Musee des Beaux Arts' 



[The students' initial interpretation before going through the 
program. ] 



This is a poem about the difference between art and ordinary life. The 'Old Masters' understand 
what is unique about life. THEY understand that suffering is a part of life. There is nothing 
unique about it. The aged sitand wait for something to occur, and children dont care one way or 
the other. Poets never forget thst while dreadful/glorious things might happen at the same time. 

In the second stanza Auden gives an illustration of what he means. Daedalus and 
Icarus are flying over the sea, an incredible event, beyond all human comprehension. Icarus flies 
close to the sun and bums his wings falling into the sea. The ploughman must have heard the 
splash, but it meant nothing to him. What is important to him is the rainfall, the amount of 
sunlight his crops will get. The rich people on the ship actually the boy go into the sea, but they 
went on their way too busy to stop. 




ee 



[The chronological log, which we use to see how the student uses 
the program, including both answers to questions and hotwords 
consulted. The latter are enclosed in brackets.] 



Musee des Beaux Arts: Log of Current Session 
The session began: Oct 17 1995 09:47 AM 

Student's Name: eugene manni 
Course and Section: ENG 42dolb 

[Masters] 

[birth] 

[martyrdom] 

[Icarus] 

[cry] 

The program was left to be finished later. 

The session ended: Oct 17 1995 10:18 AM 

Continuation of Session 

The session resumed: Oct 24 1995 09:34 AM 

Question 1 : What seems to be the subject of the painting by Bruegel, "Landscape with Fall of 
Icarus?" (1) 

Answer: 

icarus falling from the sky into the sea while others aregoing about their business. 

Question 2: Why do you think that Bruegel has not made Icarus obviously the center of attraction 
in the painting? (1) 

Answer: 

THE SUBJECT OF THE PAINTING IS NOT THE FALL OF ICARUS, BUT WHAT IS 
HAPPENING WHILE ICARUS IS FALLING. THE PLOUGHMAN IS PLOUGHING AND 
THE SHIP IS GOING ABOUT ITS BUSINESS. 

Question 3A: What is the difference between Ovid's account of these characters and the way 
Bruegel depicts them? (1) 

Answer: 

in OVID there is a detail account of how people reacted to ICARUS' FALL. 



Question 4: How does Auden describe how the characters in the painting respond to Icarus' fall? 

( 1 ) 



Answer: 

they ignore it. they go on about their business. 

Question 5: Why does he say that the fall of Icarus is "not an important failure" for the 
ploughman? (1) 

Answer: 

AN important failure for the ploughman would be if the sun didn't shine tomorrow , or if there 
was a drought so that he couldn;t feed his family. Someone falling out of the sky doesn't put food 
on the table. 

Question 6: What does the fact that the ship is "expensive" and "delicate" have to do with its 
reaction to Icarus' fall? (1) 

Answer: 

The words 'expensive' and ' delicate' give the connotation of a group of rich pampered 
individuals out for a day on their boats, away from the problems of the office or of the world 
drinking concerned with having a good time, they probably saw something, and on another day 
might have been interested by today, they have somewhere to go. 

Question 7: How would you respond if you saw "a boy falling out of the sky?" If you saw a more 
common accident, would you react the same way? (1) 

Answer: 

In all probability I would react as those on board the ship did. 

Question 8: What figures mentioned in the first part of the poem seem to exhibit the same kind of 
reaction to events as those in the second part of the poem (lines 14 to 21)? (1) 

Answer: 

Children, horses an dogs. 

Question 9: What do the miraculous birth, the dreadful martyrdom and the fall of Icarus have in 
common OTHER than the fact that to some extent they go unnoticed? (1) 

Answer: 

All thses events are beyond the normal, everyday events one lives through. They involve ideas, 
emotions beyond the ordinary. 

Question 10: Why do you think that Auden follows his introductory statements about suffering 
with the picture of the "miraculous birth," rather than the painting of the "dreadful martyrdom?" 
( 1 ) 



Answer: 

In a series birth comes first and builds to the climax of 'the dreadful martyrdom'. 

Question 1 1 : What is the part played by the non-human world (the horse, the dog, the sun) in the 
miraculous or tragic events recounted in the poem? (1) 

Answer: 

All these are either inanimate or nonrational things. Unlike humans they can not comprehend the 
significance of what Daedalus has done or what Icarus has done, these go on regardless of what 
happens about them. 

Question 12: Auden says that the Old Masters were never wrong about suffering, because they 
understood its "human position." What does he mean by "human position?" (1) 

Answer: 

To be human is to suffer. The poet knows that suffering is part of life, and no amount of feeling, 
lovecan change that. Nothing can keep us from dying, getting sick or being hurt. 

Question 13: Are Auden and Bruegel saying the same thing? (1) 

Answer: 

yes 

Question 13: Are Auden and Bruegel saying the same thing? (2) 

Answer: 

yes 

Question 13 A: What are Auden and Bruegel both saying? (1) 

Answer: 

Suffering is a part of life. The poet and the artist differ from the writer of sentimental cards in 
understanding life. 

Question 14: In what way may the meaning of the poem have been influenced by the events that 
were taking place in Europe at the time Auden wrote it? (1) 

Answer: 

In 1938, 

The program was left to be finished later. 

The session ended: Oct 24 1995 10:17 AM 
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Continuation of Session 



The session resumed: Oct 31 1995 09:31 AM 

Question 14: In what way may the meaning of the poem have been influenced by the events that 
were taking place in Europe at the time Auden wrote it? (2) 

Answer: 

In 1938 , Hitler was in power in Germany and the world was close to war. Auden had left 
England to live in America so that as a writer he could continue to produce poems. The artist 
must go on with his work regardlesss of what is happening in the world 

Question on Interpretation: 

Now that you have answered the questions, please write a short interpretation of the poem as a 
whole. Then, show to what extent the allusions in the poem helped communicate its meaning. 

Answer: 

The poems is intwo stanzas. The first stanza makes a makes a general statement and the second 
gives an example. The artist /poet knows a fundamental truth about life- in life there is suffering 
regardless of who you are or how good or bad , rich or poor you are. In the first quartrain, Auden 
calls artist/poet 'Old Masters'. In the second quartrain, Auden gives the reaction of two different 
groups to human suffering. The 'aged' having lived through suffering are wainting for the 
'miraculous birth' that will end suffering and bring in peace and love and rebirth, while children 
who have never sen or endurd suuffering don't particularly want any kind of change.The third 
quartrain brings back the 'Old Masters' and relates tern to children, horses and dogs as going 
about their business not involved in the disasters of the world. The tone of the poem , the choice 
of words are all unemotional, the second stanza is an example of Auden's theme. He takes 
Brueghel's "Icarus" as his example. In the painting we see only the leg of Icarus . The rest of 
him is in the sea. We see a ploughman working his field , and a ship sailing somewhere. They 
ignore Icarus' plight. They ignore his 'forsaken cry'. The ploughman had a family to worry about 
and the e had somewhere to go. sailing 'calmly on'. 

t. } 

Program completed. 

The session ended: Oct 31 1995 10:05 AM 




70 



« 



• < >-> 



[The log which is printed out for the students, arranged by 
question and recording all answers to the questions attempted. A 
copy of the student's final interpretation is provided on a 
separate sheet.] 



Student's Name: eugene manni 
Course and Section: ENG 42dolb 

"Musee des Beaux Arts": Student Log-Oct 17 1995 09:47 AM 

Question 1+ What seems to be the subject of the painting by Bruegel, "Landscape with Fall of 
Icarus?" 

Answer 1 : icarus falling from the sky into the sea while others aregoing about their 
business. + 

Question 2+ Why do you think that Bruegel has not made Icarus obviously the center of 
attraction in the painting? 

Answer 1 : THE SUBJECT OF THE PAINTING IS NOT THE FALL OF ICARUS, 
BUT WHAT IS HAPPENING WHILE ICARUS IS FALLING. THE PLOUGHMAN IS 
PLOUGHING AND THE SHIP IS GOING ABOUT ITS BUSINESS. + 

Question 3+ What human characters from Ovid's account of the fall of Icarus are included in 
Bruegel's painting (aside from Icarus himself)? 

+ 

Question 3A+ What is the difference between Ovid's account of these characters and the way 
Bruegel depicts them? 

Answer 1 : in OVID there is a detail account of how people reacted to ICARUS' FALL. + 

Question 4+ How does Auden describe how the characters in the painting respond to Icarus' fall? 

Answer 1 : they ignore it. they go on about their business. + 

Question 5+ Why does he say that the fall of Icarus is "not an important failure" for the 
ploughman? 

Answer 1 : AN important failure for the ploughman would be if the sun didn't shine 
tomorrow , or if there was a drought so that he couldn;t feed his family. Someone falling out of 
the sky doesn't put food on the table. + 

Question 6+ What does the fact that the ship is "expensive" and "delicate" have to do with its 
reaction to Icarus' fall? 
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Answer 1 : The words 'expensive' and ' delicate' give the connotation of a group of rich 

pampered individuals out for a day on their boats, away from the problems of the office or of the 
world drinking concerned with having a good time, they probably saw something, and on another 
day might have been interested by today, they have somewhere to go. + 

Question 7+ How would you respond if you saw "a boy falling out of the sky?" If you saw a 
more common accident, would you react the same way? 

Answer 1 : In all probability I would react as those on board the ship did. + 

Question 8+ What figures mentioned in the first part of the poem seem to exhibit the same kind of 
reaction to events as those in the second part of the poem (lines 14 to 21)? 

Answer 1 : Children, horses an dogs. + 

Question 9+ What do the miraculous birth, the dreadful martyrdom and the fall of Icarus have in 
common OTHER than the fact that to some extent they go unnoticed? 

Answer 1 : All thses events are beyond the normal, everyday events one lives through. 
They involve ideas, emotions beyond the ordinary. + 

Question 10+ Why do you think that Auden follows his introductory statements about suffering 
with the picture of the "miraculous birth," rather than the painting of the "dreadful martyrdom?" 

Answer 1 : In a series birth comes first and builds to the climax of 'the dreadful 
martyrdom'. + 

Question 1 1+ What is the part played by the non-human world (the horse, the dog, the sun) in the 
miraculous or tragic events recounted in the poem? 

Answer 1 : All these are either inanimate or nonrational things. Unlike humans they can 
not comprehend the significance of what Daedalus has done or what Icarus has done, these go on 
regardless of what happens about them. + 

Question 12+ Auden says that the Old Masters were never wrong about suffering, because they 
understood its "human position." What does he mean by "human position?" 

Answer 1 : To be human is to suffer. The poet knows that suffering is part of life, and no 
amount of feeling, lovecan change that. Nothing can keep us from dying, getting sick or being 
hurt. + 

Question 13+ Are Auden and Bruegel saying the same thing? 

Answer 1 : yes 




72 



I « 



J 



Answer 2: yes + 

Question 13A+ How do Auden and Bruegel differ? 

Answer 1 : Suffering is a part of life. The poet and the artist differ from the writer of 
sentimental cards in understanding life. + 

Question 14+ In what way may the meaning of the poem have been influenced by the events that 
were taking place in Europe at the time Auden wrote it? 

Answer 1: In 1938, 

Answer 2: In 1938 , Hitler was in power in Germany and the world was close to war. 
Auden had left England to live in America so that as a writer he could continue to produce 
poems. The artist must go on with his work regardlesss of what is happening in the world + 
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Appendix G 
Conference Program 
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LUNCH: Conferees are invited to sit in our special waterfront 3 . 3.30 p>m . 

section of the Kingsborough Community College cafeteria. Closing Remarks (Jocelyn Camp) 





Appendix H 

Software 

"Design" 

"Musee des Beaux Arts" 
"The Windhover" 
"Poetry to the Ear" (excerpts) 



[Please note: Software will follow under separate cover. | 
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